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I  SUMMARY  . 


The  goal  of  this  project  is  to  create  an  environment  suitable  for 
computer  research  activities  in  the  understanding  and  in  the  development 
of  methods  for  information  processing.  In  particular  we  will  study  computer 
network  behavior  within  the  ARPA  experimental  computer  network.  Our  studies 
include  the  mathematical  modelling  and  analysis  of  the  behavior  of  computer 
systems  emphasizing  time-shared  cortputers  and  computer  networks.  We  also 
seek  to  validate  the  results  of  such  modelling  through  the  use  of  measure¬ 
ment  procedures  and  will  serve  as  the  network  measurement  center  in  the 
ARPA  network. 


In  September,  1969,  UCLA  became  the  first,  node  in  the  ARPA  experimental 
computer  network.  This  took  place  when  we  re< eived  the  special  purpose 
message  switching  computer  (interface  message  process or-IMP) .  Within  days 
after  the  arrival  of  the  IMP,  bits  and  messages  were  being  transferred 
between  the  IMP  and  the  UCLA  Host  computer,  tie  XDS  Sigma-7.  Since  then, 
the  network  has  gram  to  include  four  nodes  (ICLA,  SRI,  UCSB,  and  the 
University  of  Utah) ;  late  in  October,  the  first  Host-Host  messages  were 
transmitted  between  UCLA  and  SRI,  marking  a  major  milestone  in  the  develop¬ 
ment  of  the  ARP?  computer  network. 

UCLA,  which  is  to  act  as  the  network  measurement  center  has  contributed 
to  development  of  the  programs  which  now  funciion  within  the  IMP  for  measui  e- 
ment  behavior.  Simple  measurements  have  already  been  performed  which  derror  - 
strafe  the  operation  and  the  usefulness  of  those  measurements  techniques. 

It  is  expected  that  this  effort  v;ill  bring  about  i aderstanding  and  insight 
into  the  network  operation  and  behavior  during  the  next  reporting  rorirvi. 
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Moreover,  it  is  an  ideal  tool  for  validating  the  mathematical  modelling 
analysis  work  to  which  we  are  devoted. 

Progress  in  the  area  of  mathematical  modelling  and  analysis  of  coipute 
systems  has  been  significant.  A  number  of  pajors  have  been  submitted, 
presented,  and  published  in  this  area  and  these  are  listed  as  references 
below.  The  effort  has  been  directed,  mainly,  in  two  areas:  time-shared 
computer  systems  analysis;  and  computer  network  analysis.  Research  results 
in.  the  former  area  have  led  to  the  beginning  of  a  caiprehensive  theory  for 
time-shared  scheduling  algorithms  find  their  analyses.  These  results  have 
been  submitted  and  accepted  to  the  highly  respected  Sixth  International 
Tele  traffic  Congress  to  be  held  in  Munich,  Germany,  September  1970  [Ref.  8] 
This  work  has  progressed  so  well,  that  it  is  now  time  to  direct  efforts 
in  attempting  to  model  other  aspects  of  time-shared  conputer  operations, 
such  as:  memory  hierarchy  structure;  paging  effects  on  the  scheduling 
algorithm;  and  other  congestion  points  such  as  input/output  congestion. 

The  second  area  of  theoretical  results  has  be  in  in  Computer  Networks .  In 
the  body  of  this  report  we  include  the  paper  "Analytic  and  Simulation 
Methods  in  Computer  Network  Design"  which  has  been  submitted  and  accepted 
for  presentatioi  and  publication  at  the  forth  rating  Spring  Joint  Catpuber 
Conference  in  Atlantic  City,  New  Jersey.  The  session  at  which  this  paper 
will  bo.  presented  is  to  be  devoted  entirely  to  the  ARPA  Computer  Network , 
and  will  undoubtedly  be  care  the  set  of  papers  most  referenced  with  regard 
to  the  ARPA  Ivetv’ork.  The  main  thrust  of  this  paper  is  that  both  analytic 
and  simulation  methods  have  been  extremely  effective  in  predicting  network 
behavior  and  ha/e  lead  to  realistic  models  of  the  ARPA  network. 
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In  addition  to  our  principal  roles  as  first  node  in  the  ARPA  network, 
Network  Measurement  Center,  and  Ccnputer  Systems  Modelling  Research  Center, 
we  have  also  been  active  in  establishing  the  j important  standards  and  proced  ires 
for  carrying  out  Host-Host  canurunication  through  appropriate  protocol  and 
languages,  This  effort  has  been  continuing  s:.nce  the  earliest  discussions 
of  the  ARPA  Network,  and  has  resulted  in  a  pa;>er  "Host-Host  Caimunication 
Protocol  in  the  ARPA  Network"  which  also  has  l>een  accepted  for  presentation 
and  publication  at  the  Spring  Joint  Computer  Conference  in  the  session 
devoted  to  the  ARPA  Network,  The  contents  of  that  paper  are  included 
also  in  the  body  of  this  report  following.  The  concepts  put  forth  in  that 
work  represent  the  results  of  many  people  in  addition  to  those  of  the  author  s/ 
and  it  is  important  to  convent  that  an  unusually  effective  association  and 
interaction  has  been  set  up  among  many  network  sites,  wherein  concerned  parties 
have  cooperated  to  create  the  network  protocol  described  in  that  paper.  Tils 
same  kind  of  cooperation  has  resulted  in  many  benefits  to  the  development  cf 
the  network,  and  we  feel  that  the  esprit  de  corps  of  this  ARPA  community  is  an 
extremely  valuable,  albeit  intangible^  asset.  The  significant  aspects  of  tnis 
protocol  paper  involve  design  concepts  for  that  protocol  within  the:  network  . 
System  calls  and  control  ccrtmands  are  defined  and  suggestions  are  made  regerd- 
ing  the  user  level  languages.  A  number  of  problems  are  solved  through  the 
standards  set  forth  in  this  paper,  but  as  is  to  be  expected,  many  more  are 
created  which  cs  yet  need  resolution  frem  among  the  users  of  this  network. 
Clearly  growth  in  this  area  must  continue  and  will  follow  with  this  paper  as 
the  major  point,  of  departure. 

The  response  from  the  computer  community  to  the  activities  taking  place 
at  UCLA  has  been  more  than  encouraging.  We  have  achieved  recognition  as  one 
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of  the  leading  carputer  system  modelling  an!  analyses  centers  in  the  t/orld. 

Our  efforts  in  measurement  of  the  network  behavior  have  stirred  up  considerable 
interest.  The  impact  of  the  partial  solutions  offered  to  the  Host-Host 
Protocol  and  Language  problem  are  just  new  beginning  to  be  felt. 

-v  .  ■  -  .  J  '  / 

II  TECHNICAL  REPORT 

Among  the  numerous  areas  of  investigation  carried  o>ut  during  this  report¬ 
ing  period,  we  choose  to  elaborate  upon  two  in  this  report.  The  first  dwells 
on  analytic  and  simulation  models  suitable  for  carputer  network  design,  as  mo¬ 
tioned  in  the  summary.  This  paper  follows  as  Section  II.  1  and  is  mode  up  of 
the  paper  submitted  to  the  Spring  Joint  Corpus  or  Conference. 

The  second  effort  emphasized  here  is  that  of  the  HOST-HOST  Ccmnunicati  an 

(4 

Protocol  in  the  AFPA  Network, and  is  presented  as  Section  II. 2,  also  in  the 
form  of  the  submitted  paper  for  the  Spring  Jo: nt  Carputer  Conference. 

Each  paper  contains  its  cwn  reference  list  and  is  thereby  self-contained. 
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ABSTRACT 

This  paper  addresses  itself  to  problems  and  solutions  in  the  mathe¬ 
matical  analysis  and  simulation  of  conputer  networks.  A  framework  is  con¬ 
structed  around  which  a  useful  theory  of  computer  networks  can  be  developed. 
The  results  so  far  obtained  provide  meaningful  insight  and  useful  aids  in 
analysis  and  design  of  these  systems.  The  ARPA  experimental  computer  network 
is  used  as  an  exanple  against  which  the  methods  of  this  paper  can  be  compared . 

ihe  paper  divides  in  three  parts .  Ihe  first  creates  a  mathematical 
queueing  model  whJ  ch  is  then  analyzed  to  yield  she  average  message  delay  fcr 
messages  travelling  through  the  network.  Tnese  analytic  computations  are  then 
compared  to  simulation  results  for  the  ARPA  computer  network  in  a  given  con- 
^  figuration;  a  model  is  found  for  which  the  agreement  between  theory  and  simu- 
#  lation  is  amazingly  good.  Ihe  second  part  addresses  itself  to  the  synthesis 
and  optimization  quesoion;  this  requires  the  definition  of  an  appropriate 
cost  function  for  the  network  and  we  carefully  examine  a  variety  of  such  cost, 
functions  which  resemble  available  data  on  commercial  transmission  systems. 


*lhis  work  was  supported  by  the  Advanced  Research  Projects  Agency  of  the 
Department  of  Defense  (DAH15-69-C-0285) . 
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The  optimization  then  reduces  to  finding  that  distribution  of  channel  capacity 
within  the  network  which  minimizes  the  average  message  delay  at  a  fixed  net¬ 
work  cost.  These  optimal  designs  are  then  compared  on  the  basis  of  average 
message  delay,  system  cost,  and  throughput  data  rate.  This  comparison  shows 
that  the  particularly  simple  linear  cost  function  (which  is  well  understood 
and  oaoy  tc  solve)  approximates  a  much  more  complicated  (albeit  more  realistic) 
cost  function,  namely  the  pcv/er  law  cost  function.  The  fact  that  the  power 
law  case  can  be  well  approximated  by  the  linear  case  is  most  valuable  since 
the  linear  case  yields  completely  to  analytic  methods  in  solving  for  the  opti¬ 
mal  distribution  of  capacity  in  networks.  The  third  part  considers  some  aspects 
of  the  operating  procedure  within  a  computer  network.  In  particular,  the 
inport  ant  question  of  how  one  should  modify  and  update  the  network  routing 
procedure  is  considered.  It  is  shown  from  simulation  that  for  the  ARPA  network 
an  asynchronous  me  thod  for  updating  is  superior  to  the  synchronous  method  in 
that  it  provides  smaller  average  message  delays;  however,  the  cost  for  asyn¬ 
chronous  updating  has  yet  to  be  accounted  for  a  id  the  software  overhead  for 
this  method  must  be  studied  in  terms  of  its  effect  on  message  delay  and  through¬ 
put.  It  is  also  shown  that  the  synchronous  updating  method  includes  transient 
looping  effects  which  if  removed  can  provide  reiuced  message  delays  as  well. 

The  results  obtained  so  far  are  most  encouraging  and  it  is  vital  that 
these  methods  be  extended  to  consider  other  performance  measures  and  network 
parameters  so  as  to  sharpen  these  already  useful  tools. 


ANALYTIC  AND  SII'iULATIQN  KbTHGDB 
IN  COMPUTED  NhT/JOHK  LESION  * 


by 

Leonard  Kleinrock 


INTRODUCfiTON 

The  Seventies  are  here  and  so  are  computer  networks!  The  tin.  sharing 
industry  dominated  the  Sixties  and  it  appears  that  computer  networks  will, 
play  a  similar  role  in  the  Seventies.  The  need  has  now  arisen  for  many  of 
these  time  shared  systems  to  share  each  others  1  resources  by  coupling  then! 
together  over  a  communication  network  thereby  creating  a  cornputer  network. 
The  mini-computer  will  serve  an  iniportant  role  nere  as  the  sophisticated 
terminal  as  well  as,  perhaps,  the  message  switching  computer  in  our  networks. 

It  is  fair  to  say  that  the  computer  industry  (as  is  true  of  most  other 
large  Industries  in  their  early  development)  has  been  guilty  of  "leaping 
before  looking”;  on  the  other  hand  ’’losses  due  to  hesitation”  are  not 
especially  prevalent  in  this  industry.  In  any  case,  it  is  clear  that  much 
is  to  be  gained  by  an  appropriate  mathematical  analysis  of  performance  and 
cost  measures  for  these  large  system's,  and  that  these  analyses  should  most 
profitably  be  undertaken  before  major  design  commitments  are  made.  This 
•  paper  attempts  tc  move  in  the  direction  of  providing  some  tools  for  and 
insight  into  the  design  of  computer  networks  through  mathematical  modeliag, 
analysis  and  simulation.  Frank,  et  al.  describe  tools  for  obtaining  low 
cost  networks  by  choosing  among  topologies  using  computationally  efficient 
methods  from  network  flow  theory;  our  approach  co:..:d omenta  theirs  in  that  we 
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look  for  closed  analytic  expressions  where  possible.  Our  intent  is  to  provide 
urders^  siding  of  the  behavior  and  trade-offs  available  in  some  computer  net¬ 
work  situations  thus  creating  a  qualitative  tool  for  choosing  design  options 
and  not  a  numerical  tool  for  choosing  precise  design  parameters. 

THE  ARPA  EXPERIMENTAL  COMPUTER  NETWORK  -  AN  EXAMPLE 

The  particular  netv;ork  which  we  shall  use  for  purposes  of  example  (and 
with  which  v:e  are  most  familiar)  is  the  Defense  Departments  Advanced  Research 
Projects  Agency  (ARPA)  experimental  computer  network.  The  concepts  basic  to 
this  network  were  clearly  stated  in  Reference  11  by  L.  Roberts  of  the  Advanced 
Research  Projects  Agency,  who  originally  conceived  this  system.  Reference  6. 
which  appears  in  these  proceedings,  provides  a  description  of  the  historical 
development  as  well  as  the  structural  organization  and  implementation  of  the 
ARPA  network.  We  choose  to  review  some  of  that  description  below  in  order  to 
provide  the  reader  with  the  motivation  and  understanding  necessary  for  main¬ 
taining  a  certain  degree  of  self  containment  in  this  paper. 

As  might  be  expected,  the  design  specifications  and  configuration  of  the 
ARPA  network  have  changed  many  times  since  its  inception  in  19 67.  in  June, 

o 

1969,  this  author  published  a  paper  in  which  a  particular  network  configun- 
tion  was  described  and  for  which  certain  analytical  models  were  constructed 
and  studied.  That  network  consisted  of  nineteen  nodes  in  the  continental 
United  States.  Since  then  this  number  has  changed  and  the  identity  of  the 
nodes  has  changed  and  the  topology  has  changed,  and  so  on.  The  paper  by 
Prank,  et  al.,  published  in  these  proceedings,  describes  the  behavior  and 
topological  desijTi  of  one  of  these  newer  versions.  V  .ever,  in  order  to  be 
consistent  with  our  earlier  results,  and  since  the  ARPA  cxomnle  is  intended 
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as  an  illustration  of  an  approach  rather  than  a  precise  design  computation, 
we  choose  to  continue  to  study  and  therefore  to  describe  the  original  nine¬ 
teen  node  network  in  this  paper. 

The  network  provides  s t  ore-and- forward  communication  paths  between  the 
set  of  nineteen  computer  research  centers.  The  computers  located  at  the 
various  nodes  are  drawn  from  a  variety  of  manufacturers  and  are  highly  incom¬ 
patible  both  in  hardware  and  software;  th?s  in  fact  presents  the  challenge  of 
the  network  experiment,  namely,  to  provide  effective  communication  among  and 
utilization  of  this  collection  of  incompatible  machines.  The  purpose  is 
fundamentally  for  resource  sharing  where  the  resources  themselves  are  highly 
specialized  and  take  the  form  of  unique  hardware,  programs,  data  bases,  and 
human  talent.  For  example,  Stanfoid  Research  Institute  will  serve  the  func¬ 
tion  of  network,  librarian  as  well  as  provide  an  efficient  text  editing  system; 
the  University  of  Utah  provides  efficient  algorithms  for  the  manipulation  of 
figures  and  for  picture  processing;  the  University  of  Illinois  will  provide 
through  its  ILLIAC  IV  the  power  of  its  fantastic  parallel  processing  capability; 
UCLA  will  serve  as  network  measurement  center  and  also  provide  mathematical 
models  and  simulation  capability  for  network  and  time-shared  system  studies. 

The  example  set  of  nineteen  nodes  is  shown  in  Figure  1  (as  of 
Spring  1569) .  The  traffic  matrix  which  describes  the  message  flow 

required  between  various  pairs  of  nodes  is  given  in  Reference  8  and  will  not 
be  repeated  here.  An  underlying  constraint  placed  upon  the  construction  of 
this  network  was  that  network  operating  procedures  would  not  interfere  in 
any  significant  way  with  the  operation  of  the  already  existing  facilities 
which  were  to  be  connected  together  through  this  network.  Consequently  the 
message  handling  tasks  (relay,  acknowledgment,  routing,  buffering,  etc.)  are 
carried  out  in  a  special  purpose  Interface  Message  Processor  (IMP)  co-located 
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with  th 3  principal  computer  (denoted  HOST  computer)  at  each  of  the  computer 
research  centers.  The  conmunication  channels  are  (in  most  cases)  50  kilobit 
pe:  second  full  duplex  telephone  lines  and  only  the  I  [-IPs  are  connected  to 
these  lines  throu,^i  data  sets.  IHus  the  communication  net  consists  of  the 
lines,  the  IKPs  and  the  data  sets  and  serves  as  the  store-and-forward  system 
for  the  HOST  comp  iter  network.  Messages  which  flow  betv;een  HOSTs  are  broken 
up  into  small  entities  referred  to  as  packets  (each  of  maximum  size  of 
approximately  100)  bits).  The  HIP  accepts  up  to  eight  of  these  packets  to 
create  a  maximum  size  message  from  the  HOST.  The  packets  make  their  way 
individually  throigh  the  IMP  network  where  the  appropriate  routing  procedure 
directs  the  traffic  flow.  A  positive  acknowledgment  is  expected  within  a 
given  time  period  for  each  inter-IMP  packet  transmission;  the  absence  of  an 
acknowledgment  forces  the  transmitting  IMP  to  repeat  the  transmission 
(perhaps  over  the  same  channel  or  sane  other  alternate  channel).  An  acknow¬ 
ledgment  may  not  oe  returned  for  example,  in  the  case  of  detected  errors  or 
for  lack  of  buffe-  space  in  the  receiving  IMF.  V/e  estimate  the  average 
packet  size  to  be  560  bits;  the  acknowledgment  length  is  assumed  to  be  1*J0 
bits.  Thus,  if  v;a  assume  that  each  packet  transmitted  over  a  channel  causes 
the  generation  of  a  positive  acknowledgment  packet  (the  usual  case,  hopefullv), 
then  the  average  packet  transmission  over  a  line  is  of  size  350  bits.  Much 
of  the  short  interactive  traffic  is  of  this  nature.  We  also  anticipate  mes¬ 
sage  traffic  of  nuch  longer  duration  and  we  rener  to  this  as  multi-packet 
traffic.  The  average  input  data  rate  to  the  e:tire  net  is  assumed  to  be  2.21 
kilobits  per  second  and  again  the  reader  is  referred  to  Reference  8  for 
further  details  c-f  this  traffic  distribution. 


So  much  for  the  description  of  the  ARPA  network.  Protocol  and  operatirg 


procedures  for  the  ARPA  computer  network  are  described  in  References  1  and  6 
in  these  proceedings  in  much  greater  detail.  The  history,  development,  moti¬ 
vation  and  cost  of  this  network  is  described  by  its  originator  in  Reference 
12.  let  us  no w  pioceed  to  the  mathematical  modelling,  analysis  and  simulation 
of  such  networks. 

ANALYTIC  AND  -SIMULATION  METHODS 

The  mathematical  tools  for  computer  network  design  are  currently  in  the 
early  stages  of  development.  In  many  ways  v:e  a^e  still  at  the  stage  of  at¬ 
tempting  to  create  computer  network  models  which  contain  enough  salient 
features  of  the  network  so  that  behavior  of  such  networks  may  be  predicted 
from  the  model  behavior. 

In  this  section  we  begin  with  the  problem  of  analysis  for  a  given  net¬ 
work  structure,  first  we  review  the  author’s  eirlier  analytic  model  of  com¬ 
munication  networks  and  then  proceed  to  identify  ttose  features  which  dis¬ 
tinguish  computer  networks  from  strict  communication  networks.  Some  previously 
published  results  on  computer  networks  are  reviewed  and  then  new  improve  ents 
on  these  results  are  presented. 

We  then  consider  the  synthesis  and  optimization  question  for  networks. 

We  proceed  by  first  discussing  the  nature  of  tie  channel  cost  function  as 
available  under  p:x?sent  tariff  and  charging  structures.  V.’e  consider  a  number 
of  different  cost  functions  which  attempt  to  approximate  the  true  data  and 
derive  relationships  for  optimizing  the  selection  of  channel  capacities  under* 
these  various  cost  functions.  Comparisons  among  the  optimal  solutions  arc 
then  made  for  the  ARPA  network. 


Finally  in  this  section  we  consider  the  operating  rules  for  computer 


networks.  V/e  present  the  results  of  simulation  for  the  ARPA  network  regard¬ 
ing  certain  aspects  of  the  routing  procedure  which  provide  improvements  in 
performance . 


A  Model  from  Queueing  Theory  -  Analysis 


In  a  recent  work  this  author  presented  some  corrputer  network  models 
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which  were  derived  from  his  earlier  research  oi  corrmunication  networks  . 

An  attempt  was  made  at  that  tire  to  incorporate  many  of  the  salient  features 
of  the  ARPA  network  described  above  into  this  conputer  network  model.  It 
was  pointed  out  that  computer  networks  differ  ft’om  communication  networks  as 
studied  in  Reference  7  in  at  least  the  following  features:  (a)  nodal  storage 
capacity  is  finite  and  may  be  expected  to  fill  occasionally;  (b)  channel  and 
modem  errors  occur  and  cause  re-transmission;  (<:)  acknowledgment  messages  in¬ 
crease  the  message  traffic  rates;  (d)  messages  Irom  HOST  A  to  HOST  B  typically 
create  return  traffic  (after  some  delay)  from  B  to  A;  (e)  nodal  delays  become 
important  and  comr. arable  to  channel  transmission  delays;  (f)  channel  cost 
functions  are  more  conplex.  V/e  intend  to  inclule  some  of  these  features  in 
our  model  below. 

The  model  proposed  for  computer  networks  is  drawn  from  our  corrmunicat.* on 
network  experience  and  includes  the  following  assumptions .  V/e  assume  that 
the  message  arrivals  form  a  Poisson  process  with  average  rates  taken  from  a 
given  traffic  matrix  (such  as  in  Reference  8),  where  the  message  lengths  are 
exponentially  distributed  with  a  mean  1/p  of  350  bits  (note  that  we  are  only 
accounting  for  short  messages  and  neglecting  the  multi-packet  traffic  in 
this  model).  As  discussed  at  length  in  Rcferer.ce  7,  we  also  make  the 
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Independence  assumption  which  allows  a  very  simple  node  by  node  aria lysis. 

We  further  assume  that  a  fixed  routing  procedure  exists  (that  is,  a  unique 
allowable  path  exists  from  origin  to  destination  for  each  origin-destination 
pair’) . 

Prom  the  above  assumptions  one  may  calculate  the  average  delay  due  to 
waiting  for  and  transmitting  over  the  ith  channe  l  from  Eq.  (1), 


(1) 


where  X^  is  the  average  number  of  messages  per  second  flowing  over  channel  i 

(whose  capacity  is  bits  per  second) .  This  was  the  appropriate  expression 

7 

for  the  average  channel  delay  in  the  study  of  cannunication  riots  and  in  that 
study  we  chose  as  our  major  performance  measure  the  message  delay  T  averaged 
over  the  entire  network  as  calculated  from 

1'  =1  ~  T  (2) 

i  ^  1 

where  y  equals  the  total  inpuc  data  rate.  Note  that  the  average  on  is 
'  formed  by  weighting  the  delay  on  channel  Cb  with  the  traffic,  X^,  carried  on 
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that  channel.  In  tte  study  of  communication  necs  this  last  equation  provided 
an  excellent  means  for  calculating  the  average  message  delay.  That  study 
went  on  to  optimise  the  selection  of  channel  capacity  throughout  the  network 
under  the  constraint  of  a  fixed  cost  winch  was  assumed  to  be  linear  with 
capacity;  we  elaborate  upon  this  cost  function  later  in  tins  section. 
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The  computer  network  models  studied  in  Reference  8  also  made  use  of 
Eq.  (1)  for  the  calculation  of  the  cliannel  delays  (including  queueing)  where 
parameter  choices  were  1/p  =  350  bits,  =  50  kilobits  and  X ^  -  average  mes¬ 
sage  rate  on  channel  i  (as  determined  from  the  traffic  matrix,  the  routing 
procedure ,  and  accounting  for  the  effect  of  acknowledgment  traffic  as  men¬ 
tioned  in  feature  (c)  above).  In  order  to  account  for  feature  (e)  above,  the 
performance  measure  (taken  as  the  average  message  delay  T)  was  calculated 
from 


where  again  y  =  total  inpui,  data  rate  and  the  term  10  =  1  millisecond 

(nominal)  is  included  to  account  for  the  assume  1  (fixed)  nodal  processing 
time.  The  result  of  this  calculation  for  the  ARPA  network  shown  in  Figure  1 
may  be  found  In  Reference  8. 

The  computer  network  model  described  above  is  essentially  the  one  used 
for  calculating  dela  s  in  the  topological.  studi2s  reported  upon  by  Frank  et 
al.  in  these  proceedings. 

A  number  of  simulation  experiments  have  been  carried  out  using  a  rather 
detailed  descript-.cn  of  the  ARPA  network  and  its  operating  procedure.  Some 
of  these  results  were  reported  upon  in  Reference  8  and  a  comparison  was  made 
there  between  the  theoretical  results  obtained  from  Eq.  (3)  and  the  simulation 
results.  This  comparison  is  reproduced  in  Figure  ?  where  the  lowest  curve 
corresponds  to  the  results  of  Eq,  (3) .  Clear  1\  the  comparison  between  simula¬ 
tion  and  trieory  is  only  mildly  satisfactory.  As  pointed  out  in  Reference  8, 
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the  discrepancy  is  clue  to  the  fact  tiiat  the  acknowledgment  traffic  has  been 
improperly  included  in  Equation  3»  An  attempt  was  made  in  Reference  8  to 
properly  account  for  the  acknowledgment  traffic ;  however,  this  adjustment 
was  unsatisfactory.  The  problem  is  tiiat  the  average  message  length  has  been 
taken  to  be  350  bits  and  this  length  has  averaged  the  traffic  due  to  acknowl¬ 
edgment  messages  along  with  traffic  due  to  real  messages.  These  acknowledg¬ 
ments  should  not  be  included  among  those  messages  whose  average  system  delay 
is  being  calculated  and  yet  acknowledgment  traffic  must  be  included  to  properly 
account  for  the  true  leading  effect  in  the  network.  In  fact,  the  appropriate 
way  to  include  this  effect  is  to  recognize  that  the  time  spent  waiting  for  a 
channel  is  dependent  upon  the  total  traffic  (including  acknowledgments )  whereas 
the  time  spent  in  transmission  over  a  channel  should  be  proportional  to  the 
message  length  of  the  real  message  traffic.  Moreover,  our  theoretical  equa¬ 
tions  have  accounted  only  for  transmission  delays  which  come  about  due  to  the 
finite  rate  at  which  bits  may  be  fed  into  the  charnel  (i.e.,  50  kilobits  per 
second);  we  are  required  however  to  include  also  the  propagation  time  for  a 
bit  to  travel  down  the  length  of  the  channel.  Lastly,  an  additional  one 
millisecond  delay  is  included  in  the  final  destination  node  in  order  to  deliver 
the  message  to  tin  destination  HOST.  These  additional  effects  give  rise  to  the 
following  expression  for  the  average  message  delay  T. 

T  =£  T  ( +  +  PLi + 10-3 )  + 10-3  (i,) 

where  1/jj'  =  560  bits  (a  real  message's  average  length)  and  PL^  is  tiie  propa¬ 
gation  delay  (dependent  on  the  channel  length,  1.^)  for  the  i^  channel.  The 
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first  term  in  parentheses  is  the  average  transmission  time  and  the  second 
tern  is  tlie  averag:  waiting  time .  The  result  of  this  calculation  for  the 
ARPA  network  gives  us  the  curve  in  Figure  2  labelled  "theory  with  correct 
acknowledge  adjustment  and  propagation  delays."  The  correspondence  now 
between  simulation  and  theory  is  unbelievably  good  and  we  are  encouraged  that 
this  approach  appeal's  to  be  a  suitable  one  for  t  he  prediction  of  computer  net¬ 
work  performance  for  the  assumptions  made  lie  re.  In  fact,  one  can  go  further 
and  include  the  effect  on  message  delay  of  the  priority  given  to  acknowledg¬ 
ment  traffic  in  the  ARPA  network;  if  one  includes  this  effect,  one  obtains 
another  excellent  fit  to  the  simulation  data  labelled  in  Figure  2  as  "theory 
corrected  and  with  priorities." 

As  discussed  in  Reference  3  one  may  generalize  the  model  considered 
herein  to  account  for  more  general  message  length  distributions  by  making  use 
of  the  Pollaczek-KhincMn  formula  for  the  delay  T  of  a  channel  with  capacity 

p 

where  the  message  lengths  have  mean  l/\i  bits  with  variance  o  ',  where  Ai 
is  the  average  message  traffic  rate  and  p.^  =  A^/pC^  which  states 


.  Pi(l  + 

*■  2n^~-—.T 


This  expression  wc-uld  replace  the  first  two  terns  in  the  parenthetical  cxnrer - 
sion  of  Eq.  (*l);  of  course  by  relaxing  the  assumption  of  an  exponential  distri¬ 
bution  we  remove  ilie  simplicity  provided  by  the  Markovian  property  of  the 
traffic  flow.  This  approach,  however ,  should  provide  a  better  approximation 
to  the  true  behavior  when  required. 
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Having  briefly  considered  the  problem  of  analyzing  computer  networks 
with  regard  to  a  single  performance  measure  (average  message  delay),  we  now 
move  on  to  the  consideration  of  synthesis  questions.  This  investigation 
ininediately  leads  into  optimal  synthesis  procedures. 


Optimization  for  Various  Channel  Cost  Functions — Synthesis 
We  are  concerned  here  with  the  optimization  of  the  channel  capacity 
assignment  under  various  assumptions  regarding  wire  cost  of  these  channels. 
This  optimization  must  be  made  under  the  constrnint  of  fixed  cost.  Our  prob¬ 
lem  statement  'then  becomes:* 

Select  the  {C. }  so  as  to  minimize  T 

1  (6) 


subject  to  a  fixed  cost  constraint 
where,  for  simplicity,  we  use  the  expression  in  Eq.  (2)  to  define  T. 

We  are  now  faced  with  choosing  an  appropriate  cost  function  for  the 
system  of  channels.  Vie  assume  tint  the  total  cost  of  the  network  is  contained 
in  these  channel  costs  where  we  certainly  permit  fixed  termination  charges, 
for  example,  to  be  included.  In  order  to  get  a  feeling  for  tlxe  correct  form 
for  the  cost  function  let  us  examine  seme  available  data.  From  Reference  3 
we  have  available  the  costing  data  which  we  present  in  Table  1.  Frcm  a  sched¬ 
ule  of  costs  for  leased  communication  lines  available  to  ARPA  we  have  the  data 
presented  in  Table  2. 


*The  dual  to  this  optimization  problem  may  also  be  considered:  ’’Select 
the  {Cb}  so  as  to  minimize  cost,  D,  subject  to  a  fixed  message  delay  con¬ 
straint.”  The  solution  to  this  dual  problem  gives  the  optimum  with  the 
same  A;ncticnal  dependence  on  as  one  obtains  for  the  original  optimization 
problem. 


11 


TABIjE  I — 

■Publicly  available 
line  costs  from 

leased  transmission 
Reference  3 

Speed 

[ 

Cost/mi  le/monti  i 
(normalized  to 
1000  mile  distance) 

^5 

bps 

$  .70 

56 

bps 

.7° 

75 

bps 

.77 

2^00 

bps 

1.79 

Hi 

KB 

15.00 

82 

KB 

20.00 

230 

KB 

28.00 

1 

MB 

60.00 

12 

MB 

287.50 

TABLE  II— Estimated  leased  transmission  line  costs  based  on  Telpak  rates 


Cost  Cost/to  le/month 

(termination  +  mileage)  (normalised  to 

Speed  _ /faonth _ ■  1000  mile  distance) 


±50 

CpS 

-4 

-4 

0 

+ 

$  .12/mile 

$  .20 

-  2^00 

bps 

232. 

+ 

.35/mile 

.58 

7200 

bps 

810 

+ 

.35/mile 

1.16 

19.2 

K3 

850 

+ 

2.10/mile 

2.95 

50 

KB 

850 

+ 

20/mile 

5.05 

108 

KB 

21)00 

+ 

k. 20/mile 

6.6c 

230.1} 

KB 

1300 

+ 

21.00/mile 

22.30 

1160.8 

KB 

1300 

+ 

60.00/mile 

61.30 

1.3HH 

MB 

50C 

+ 

75.00.4nHe 

80.00 

*Tnese  costs 

are ,  in 

some  cases, 

first  estimate 

s  and  are  not 

considered  as  quoted  rates. 
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We  have  plotted  these  functions  in  Figure  3.  We  must  now  attempt  to  find  an 
analytic  function  which  fits  cost  functions  of  this  sort.  Clearly  ttiat  ana¬ 
lytic  function  will  depend  upon  the  rate  schedule  available  to  the  computer 
network  designer  and  user.  Many  analytic  fits  to  this  function  liave  been 
proposed  and  in  particular  in  References  3  a  fit  is  proposed  of  the  form: 

Cost  of  line  =  0.1  $/mile/month  (7) 

10 

Based  upon  rates  available  for  private  line  enamels,  F>rastrcmonacolu  arrives 
at  the  following  fit  for  line  costs  where  he  has  normalized  to  a  distance  of 
50  miles  (rather  than  1000  miles  in  Eq.  (7)) 

Cost  of  line  =  1.08  C.0’^  $/mile/month  (8) 


Referring  nov;  to  Figure  3  we  see  that  the  mi.lea  ~e  costs  fretn  Table  II  rise 
as  a  fractional  e}ponent  of  capacity  (in  fact  with  an  exponent  of  .815)  sug¬ 
gesting  the  cost  Jlmction  shown  in  Eq.  (9)  bcl:w 

Cost  of  line  =  A  C^*^"  $/mile/month  (9) 


These  last  three  equations  give  trie  dollar  cost  per  mile  per  month  where  the 
capacity  C^  is  giren  in  bits  per  second.  It  interesting  to  note  that  all 
three  functions  ai'e  of  the  form 


Cost  of  line  =  A  C, 


$/mi le /month 


(10) 


It  is  clear  from  these  simple  considerations  that  the  cost  function  appropriate 
for  a  particular  application  depends  upon  that  application  and  therefore  it  is 
difficult  to  establish  a  unique  cost  function  for  all  situations.  Consequently, 


13 


we  satisfy  ourselves  below  by  considering  a  rubber  of  possible  cost  functions 
ana  study  optimization  conditions  and  results  which  follow  from  those  cost 
functions.  The  designer  may  then  choose  from  among  these  to  match  his  given 
tariff  schedule.  These  cost  functions  will  fom  the  fixed  cost  constraint 
in  Eq.  (6).  Let  us  now  considei  tlie  collection  of  cost  functions,  and  the 
related  optimization  questions. 

1.  Linear  cost  function.  We  begin  with  this  case  since  the  analysis 
already  exists  in  the  author's  Reference  7,  where  the  assumed  cost  constraint 
took  the  form 


D  =2  d,c,  (11) 

i  11 

where  D  =  total  number  of  dollars  available  to  spend  on  channels,  =  the 

1.U 

dollar  cost  per  ur.it  of  capacity  on  the  i  channel,  and  once  again  is  the 
capacity  of  the  ith  channel.  Clearly  Eq.  (11)  Is  of  the  same  fom  as  Eq.  (10) 
with  a  «  1  where  we  now  consider  the  cost  of  alL  channels  in  the  system  as 
having  a  linear  form.  Ms  cost  function  assumes  that  cost  is  strictly 

linear  with  respect  to  capacity;  of  course  this  same  cost  function  allows  the 
assumption  of  a  constant  (for  example,  termination  charges)  plus  a  linear  cost 
function  of  capacity.  This  constant  (termination  charge)  for  each  channel  may 
be  subtracts  out  of  total,  cost,  D,  to  create  an  equivalent  problem  of  the  form 
given  in  Eq.  (11).  Tine  constant,  d^,  allows  orr  to  account  for  the  length  of 
the  channel  since  ch  may  clearly  be  proportional  to  the  length  of  the  channel', 
as  well  as  anything  else  regarding  the  particular  c’nannel  involved  such  as, 


for  example,  the  terrain  over  which  the  channel  must  be  placed.  As  was  done 
in  Reference  7,  one  may  carry  out  the  minimization  giver,  by  Eq.  ((')  using,  for 


1*» 


This  procedure 


5 

exanple,  the  method  of  Ingrangian  undetermined  multipliers, 
yields  tlie  following  equation  for  the  capacity 


/M  _M_ 

\ai  f 

j 


(12; 


where 

X.d, 

ue  =  D  "  S  Ar  >  0  U3) 

i  y 

When  we  substitute  this  result  back  into  Eq.  (2)  we  obtain  that  tlie  perform¬ 
ance  measure  for  such  a  cliannel  capacity  assignment  is 


T  = 


where 

£  h 

n  =  —  =  ~~  -  average  rath  length  (15> 

Tne  resulting  ICq.  (12)  is  referred  to  as  the  square  root  channel  capacity 
assignment;  this  particular  assignment  first  provides  to  each  channel  a  capacity 
equal  to  X^/u  which  is  merely  the  average  bit  late  which  must  pass  over  that 
channel  and  which  it  must  obviously  be  prcvidec  if  tlie  cliannel  is  to  carry 
such  traffic.  In  addition,  surplus  capacity  (cue  to  excess  dollars,  D  )  is 

c 

assigned  to  this  cliannel  in  proportion  to  the  square  root  of  the  traffic  car¬ 
ried,  hence  the  name.  In  Reference  7  the  author  studied  in  great  detail  the 
particular  case  for  winch  «  1  (the  case  for  which  all  charnels  cost  the 
same  regardless  of  length)  and  considerable  information  regarding  topological- 

lb 


design  arid  routing  procedures  was  thereby  obtained.  However,  in  the  case 
of  the  ARPA  network  a  more  reasonable  choice  for  d,^  is  that  it  should  be 
proportional  to  the  length  of  the  channel  as  indicated  in  Eq.  (10) 

(for  a  =  1)  which  gives  the  per  mileage  cost;  thus  we  may  take  d.^  =  AL^. 

This  second  case  was  considered  in  Reference  8  and  also  in  Reference  9.  The 
interpretation  for  these  two  cases  regarding  the  desirability  of  concentrating 
traffic  into  a  fevf  large  and  short  channels  as  v/ell  as  minimizing  the  average 
length  of  lines  traversed  by  a  message  was  well  discussed  and  will  not  be 
repeated  here. 

We  observe  in  the  ARPA  network  example  since  the  channel  capacities  are 
fixed  at  50  kilobits  that  there  is  no  freedom  left  to  optimize  the  choice  of 
channel  capacities;  however  it  was  shown  in  Reference  8  that  one  could  take 
advantage  of  the  optimization  procedure  in  the  following  way:  The  total  cost 
of  the  network  using  50  kilobit  channels  may  be  calculated.  One  may  then 
optimize  the  network  (in  the  sense  of  minimizing  T)  by  allowing  the  channel 
capacities  to  vary  while  maintaining  the  cost  fixed  at  this  figure.  The  result 
of  such  optimizat  ion  will  provide  a  set  of  channel  capacities  which  vary  con¬ 
siderably  from  the  fixed  capacity  network.  It  was  shown  in  Reference  8  that 
one  could  improve  the  performance  of  the  network  in  an  efficient  way  by  allow- 
-  ing  that  channel  diich  required  the  largest  capacity  as  a  result  of  optimiza¬ 
tion  to  be  increased  from  50  kilobits  in  the  fixed  net  to  250  kilobits.  This 
of  course  increases  the  cost  of  the  system.  Ore  may  then  provide  a  250  kilobit 
channel  for  the  second  "most  needy"  cliannel  from  the  optimization,  increasing 
the  cost  further.  One  may  then  continue  this  procedure  of  increasing  the  needy 
channels  to  250  kilobit::  while  increasing  the  cost  of  the  network  and  observe 


the  way  in  which  message  delay  deers 


as 


stem  cost  increases.  It  wa: 
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found  that  natural  stopping  points  for  this  procedure  existed  at  which  the 
cost  increased  rapidly  without  a  similar  sharp  decrease  in  message  delay 
thereby  providing  some  handle  on  the  cost-perfo.mnance  trade-off. 

Since  we  are  more  interested  in  the  difference  between  results  obtained 
when  one  varies  the  cost  function  in  more  significant  ways,  we  now  study 
additional  cost  ft notions. 


2.  Logarithmic  cost  functions.  The  next  case  of  interest  assumes 
a  cost  function  of  the  form 

D  “  £  q  l°Se  a.  CL  (16) 

where  D  again  is  the  total  dollar  cost  provided  for  constructing  the  network, 
d^  is  a  coefficient  of  cost  which  rnay  depend  up  an  length  of  crianneT,  a  is  an 
appropriate  multiplier  end  is  the  capacity  of  the  i^h  channel.  We  consider 
this  cost  function  for  two  reasons:  first,  because  it  has  the  property  that 
the  incremental  cost  per  bit  decreases  as  the  channel  size  increases;  and 
secondly,  because  it  leads  to  simple  theoretical  results.  We  now  solve  the 
minimization  problem  expressed  in  Eo.  (6)  where  the  fixed  cost  constraint  is 
now  given  tlirough  Eq.  (16).  We  obtain  the  following  equation  for  the  capacity 
of  the  ith  clianne1. 


(17) 


In  this  solution  the  Lagrangian  multiplier  (5  must  be  adjusted  so  t;iat  Eq.  (IS) 
is  satisfied  when  is  substituted  in  from  Eq.  (17).  note  the  unusual  sim¬ 
plicity  for  tne  solution  of  Ch  ,  namely  that  the  channel  capacity  for  the  i^1^ 


37 


channel  is  directly  preport  long  1  to  the  traffic  carried  by  that  channel, 

A^/p.  Contrast  this  result  with  the  result  in  Eq.  (12)  where  we  had  a  square 
root  channel  capacity  assignment .  If  we  now  take  the  siirole  result  given  in 
Fq.  (17)  and  use  it  in  Eq.  (2)  to  find  the  performance  measure  T  we  obtain 


In  this  last  result  the  performance  measure  depends  upon  the  particular  dis¬ 
tribution  of  the  internal  traffic  {A^/p}  through  the  constant  0  which  is 
adjusted  as  described  above. 

3.  The  power  law  cost  function.  As  we  saw  in  Eqs.  (7),  (8),  and 
(9)  it  appears  that  many  of  the  existing  tariffs  may  be  approximated  by  a 
cost  function  of  the  form  given  in  Eq.  (19)  below. 


D  =  E  d,c“  (19) 

i  1  1 

where  a  is  sane  appropriate  exponent  of  the  capacity  and  d^  is  an  arbitrary 
multiplier  which  may  of  coi*rse  depend  upon  the  length  of  tlie  channel  and  other 
pertinent  channel  parameters.  Applying  the  Lajjrangian  again  with  an  undeter* 
min.  j  multiplier  0  we  obtain  as  our  condition  for  an  optimal  channel  capacit  / 
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where 


(21) 


Once  again,  3  must  be  adjusted  sc  as  tc  satisfy  the  constraint  Eq.  (19). 

It  can  be  shown  that  the  left  hand  side  of  Eq.  (20)  represents  a  convex 
function  and  that  it  has  a  unique  solution  for  some  positive  value  C^.  We 
assume  that  a  is  in  the  range 


0  <  a  <  1 


as  suggested  from  the  data  in  Figure  3.  We  may  also  show  that  the  loca¬ 
tion  of  the  solutLon  to  Eq.  (20)  is  not  especially  sensitive  to  the  parameter 
setting.  Therefore,  it  is  possible  to  use  any  efficient  iterative  teciinique 
for  solving  Eq.  (20)  and  we  have  found  that  such  techniques  converge  quite 
rapidly  to  the  optimal  solution. 


Comparison  of  solutions  for  various  cost  functions.  In  the  last 
three  subsections  v:e  have  considered  three  different  cost  functions:  the 
linear  cost  function;  the  logarithmic  cost  function;  and  the  power  lav/  cost 
function.  Of  course  we  sec  immediately  that  the  linear  cost  function  is  a 
special  case  a  =  1  of  the  power  law  cost  function.  We  wish  now  to  compare  the 
performance  and  cost  of  computer  networks  under  these  various  cost  functions. 
We  use  for  our  example  the  ARP  A  computer  network  as  described  above. 

It  is  not  obvious  how  one  should  proceed  in  making  this  comparison.  How¬ 
ever,  we  adopt  the  following  approach  in  an  attempt  to  make  seme  meaningful 


earns  risers .  We  consider  the  ARPA  network  at  a  traffic  load  of  100  j  of  the 
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full  data  rate,  namely  225  kilobits  per  second  (denoted  by  yq).  For  the 
50  kilobit  net  shovel  in  Figure  1  we  may  calculate  the  line  eosts  from  Table 
II  (eliminating  the  termination  ci&rgps  since  we  recognize  this  causes  no 
essential  change  in  our  optimization  procedures,  as  mentioned  above);  the 
resaltant  network  cost  is  approximately  $579,000  per  year  (which  we  denote 
by  Dq).  Using  this  yQ  and  DQ  (as  well  as  the  other  given  input  parameters) 
we  may  then  carry  out  the  optimization  indicated  in  Eq.  (6)  for  the  case  of 
a  linear  cost  function  where  d^  =  AL^  and  A  is  immediately  found  from  the 
mileage  cost  in  Table  II.  This  calculation  results  in  an  average  message 
delay  Tq  (calculated  from  Eq. (14) ) whose  value  is  approximately  24  milliseconds. 
We  have  now  established  an  "operating  point"  for  the  tliree  quantities  yQ,  Dq, 
and  Tq,  whose  valres  are  1002  of  full  data  rate,  $579,000,  and  24  milliseconds, 
respectively. 

We  may  now  examine  all  of  our  other  cost  functions  by  forcing  them  to 
pass  through  this  operating  point.  We  assure  d^  =  AL^  throughout  for  these 
calculations.  Also  v;e  choose  a  =  1  for  the  logarithmic  case  in  Eq.  (16).  (Note 
for  the  logarithmic  and  power  law  cases,  that  two  unknown  constants,  6  and  A, 
must  be  determine I;  this  is  now  easily  done  if  we  set  T  =  Tq  and  D  =  Dq  for 
y  -  Yq  in  each  of  these  two  cases  independently. )  In  particular  now  we  wish 
to  examine  the  behavior  of  the  network  under  these  various  cost  functions. 

Wc  do  this  first  by  fixing  the  cost  of  the  network  at  D  =  Dq  and  plotting 
T,  the  average  time  delay,  as  we  '/ary  the  percentage  of  full  data  rate  applied 
to  the  network ;  this  perfoimarice  is  given  in  Figure  4  where  we  show  the  system 
behavior  for  the  power  law  cost  function  and  tie  linear  cost  function.  Trie 
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result  is  striking!  We  see  tint  the  variation  in  average  message  delay  is 
almost  insignificant  as  a  passes  through  the  range  from  0.3  to  1.0. 

We  conclude  then  that  the  very  important  pov.w_lav:  cost  function  may  be  ana¬ 
lyzed  using  a  linear  cost  function  v;hen  one  is  interested  in  evaluating  the 
average  time  delay  at  fixed  cost . * 

We  also  consider  the  variation  of  the  network  cost  D  as  a  function  of 
data  rate  at  fixed  average  message  delay,  namely  T  =  rfQ  =  2^  milliseconds. 

This  performance  is  shown  in  Figure  5  for  all  three  cost  functions.  We  note 
here  that  the  linear  cost  function  is  only  a  fair  approximation  to  the  power  law 
cost  function  over  the  range  of  a  shown;  the  logarithmic  cost  .[Unction  is  also 
shown  and  behaves  very  much  like  the  linear  cost  function  for  data  rates  above 
Yq  but  departs  from  that  behavior  for  data  rates  below  yQ.  It  can  be  shown 
that  the  network  cost,  D,  at  fixed  T  =  ‘lj  for  the  case  a  =  1  (linear  cost  func¬ 
tion  )  varies  as  a  constant  plus  a  linear  dependence  on  y.  It  is  also  of  inter¬ 
est  to  cross  plot  the  average  time  delay  T  with  the  network  cost  D,  This  we 
do  in  Figure  6  fo'1  the  class  of  power  law  cost  functions.  In  Figures  6a  and 
6b  vie  obtain  points  along  t ixc  vertical  and  horizc  :tal  axes  corresponding  to 
fixed  delay  and  fixed  cost,  respectively.  These  loci  are  obtained  by  varying 
Y  and  we  connect  the  points  for  equal  y  with  straight  lines  as  shown  in  th* 
figure  (however,  \re  in  no  way  imply  that  the  system  passes  along  these  straight 
lines  as  both  T  aid  D  arc  allowed  to  vary  simultaneously).  We  note  the  increased 
range  of  Das  a  varies  from  0.3  to  1.0,  but  very  little  change  in  the  range  of 


*The  logarithm  cost  function  is  not  shov.ii  in  Figure 
delay  is  extremely  sensitive  to  the  data  rate  and  bears 
to  the  rower  lav:  case. 


'I  since  the  time 
]  itt'le  resemblance 
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T.  In  Figure  6c  wc  collect  together  the  behavior  3n  this  plane  for  many 
values  of  a  where  the  line;  labelled  with  a  particular  value  of  a  correspond 
to  the  502  data  rate  case  in  the  lower  left-ha  id  portion  of  the  figure  and 
to  the  1302  data  rate  case  in  the  upper  right- and  portion  of  the  figure. 

From  Figure  6c  v:e  clearly  observe  that  for  fixed  cost  the  time  delay  range 
varies  Insignificantly  as  a  changes  (as  we  emphasized  in  discussing  Figure  *1). 
Similarly,  we  observe  the  moderate  variation  at  fixed  time  delay  of  network 
cost  as  a  ranges  through  its  values  (this  we  saw  clearly  in  Figure  5). 

lhese  studies  of  network  optimization  for  various  cost  functions  need 
further  investigation.  Our  aim  in  this  section  has  been  to  exhibit  some  of 
the  performance  characteristics  under  these  cost  functions  and  to  compare  them 
in  some  meaningful  way. 


Simulated  Routing  in  the  ARPA  Network--Opc rat Ing  Procedure 
We  have  examined  analysis  and  synthesis  procedures  for  computer  networks 
above.  We  now  proceed  to  exhibit  some  properties  of  the  network  operating 
procedure,  in  particular,  t he  message  routing  procedure. 

The  ARPA  network  uses  a  routing  procedure  v/hich  Is  local  in  nature  as 
opposed  to  global.  Some  details  of  this  procedure  are  available  in  Reference  6 
in  these  procecdi  igs  and  we  -wish  to  conment  on  trie  method  used  for  updating 
the  routing  table;..  For  purposes  of  routing,  each  node  maintains  a  list  whiJh 
contains  for  each  destination  an  estimate  of  tie  delay  a  message  would  encounter 
in  attempting  to  reach  that  destination  nolo  were  it  to  be  sent  out  over  a 
particular  channel  emanating  from  that  node;  U  -  list  contains  on  entry  for 
each  destination  and  each  line  leaving  the  node  in  which  this  list  is  contained. 


Every  inlf  second  (anprox.irntoly)  each  node  sends  to  all  of  its  immediate  neigh¬ 
bors  a  list  which  contains  its  estimate  rn  t  ^  rherto'k  ri^ay  t.l*  :  to  pass  to 
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each  destination;  this  list  therefore  contains  a  number  of  entries  which  is 
one  less  than  the  number  of  nodes  in  the  network.  Upon  receiving;  this  informa¬ 
tion  from  one  of  its  neighbors  the  IMP  adds  to  this  list  of  estimated  delays  * 
a  measure  of  the  current  delays  in  passing  from  itself  to  the  neighbor  from 
whom  it  is  receiving  this  li~4  ;  this  then  provides  that  IMP  an  estimate  of 
the  minimum  delay  'required  to  reach  all  destinations  if  one  travelled  out 
over  the  line  connected  to  that  neighbor,  "he  routing  table  for  the  HIP  is 
then  constructed  by  combining  the  lists  of  all  of  its  neighbors  into  a  set 
of  columns  and  cl  nosing  as  the  output  line  for  messages  going  to  a  particular 
destination  that  line  for  which  the  estimated  delay  over  that  line  to  that 
destination  is  minimum.  What  we  have  here  described  is  essentially  a  period’s 
or  synchronous  upiating  method  for  the  routing  tables  as  currently  used  in  the 
ARPA  network.  It  has  the  clear  advantages  of  providing  reasonably  accurate 
data  regarding  path  delays  as  well  as  the  important  advantage  of  being  a 
rather  simple  procedure  both  from  an  operational  point  of  view  and  from  an 
overhead  point  of  view  in  terms  of  software  costs  inside  the  IMP  program . 

We  suggest  that  a  more  efficient  procedure  in  terms  of  routing  delays  is 
to  allow  asynchrc nous  updating ;  by  this  we  mean  that  routing  information  Is 
passed  from  a  noce  to  its  nearest  neighbors  only  when  significant  enough  changes 
Occurred  in  its  own  routing  table  to  warrant  such  an  information  exc’narge .  The 
definition  of  "significant  enough”  must  be  stulied  carefully  but  certainly 
implies  the  use  c-f  thresholds  on  the  percentage  change  of  estimated  delays. 

When  these  thresholds  are  crossed  in  an  IMP  t r sn  routing  information  is  trans¬ 


ferred  to  that  IMP’S  nearest  neighbors.  This  asynchronous  mode  of  updating 
implies  a  Large  overhead  for  updating  and  it  remains  to  be  seen  whether  the 


advantage.”  gained  through  this  moio  elaborate  updating  method  overcome  t he 
disadvantages  due  to  soffwar-e  costs  and  cycle-stealing  costs  for  updating. 

We  nay  observe  the  difference  in  performance  between  yncnronous  and  asynchro¬ 
nous  updating  through  the  use  of  simulation  as  ihown  in  Figure  7.  In  this 
figure  we  plot  the  average  time  delay  T  versus  the  average  path  length,  for 
messages  under  various  routing  disciplines.  We  observe  immediately  that  the 
three  points  shown  for  asynchronous  updating  arc  significantly  superior  to 
those  shown  for  synchronous  updating.  For  a  comparison  we  also  show  the  resilt 
of  a  fixed  routing  algorit..i  which  was  computed  by  solving  for  the  shortest 
delay  path  in  an  unloaded  net  <rk;  the  asynchronous  updating  shows  superior 
performance  to  the  fixed  routing  roceuure .  However,  the  synchronous  updating 
shows  inferior  performance  compared  to  this  very  simple  fixed  routing  procecli.ro 
if  we  take  as  our  performance  measure  the  average  message  delay. 

It  was  observed  that  with  sync) ironous  updating  it  was  possible  for  a 
message  to  get  tripped  temporarily  in  loops  (i.e. ,  travelling  back  and  forth 
between  tie  same  pair  o_  nodes)  We  suppressed  this  looping  behavior  for 
two  synchronous  updating  procedures  with  different  parameter  settings  and 
achieved  significant  improvement;  nevertheless,  this  improved  version  remain;; 
inferior  to  those  simulated  systems  with  asynchronous  updating.  As  mentioned 
al)ove,  asynclirorcus  updating  contains  many  virtues,  but  one  must  consider 
the  overhead  inci  rivd  for  such  a  sophisticated  updating  procedure  before  it 


can  be  incorporated  ar.d  expo  ted  to  yield  a  net  improvement  in  performance. 


CuilCLl  PIOUS 


Our  goal  in  this  paper  lias  been  to  demonstrate  the  importance  of  analyti¬ 
cal  and  simulation  techniques  in  evaluating  computer  networks  in  the  early 
design  stages.  We  have  addressed  ourselves  to  three  areas  of  interest, 
namely  the  analysis  of  computer  network  performance  using  methods  from  queueing 
theory,  the  optimal  synthesis  problem  for  a  variety  of  cost  functions,  and  the 
choice  of  routing  procedure  fur  these  networks.  Cur  results  show  that  it  is 
possible  to  obtain  exceptionally  good  results  in  the  analysis  phase  when  one 
considers  the  "small"  packet  traffic  only.  As  yet,  v;e  have  not  undertaken 
the  study  of  the  rnulti -packet  traffic  behavior.  In  examining  available  data 
we  found  tint  the  power  lav;  cost  function  appears  to  be  the  appropriate  one 
for  high-speed  data  linens.  We  obtain  optimal  aiannel  capacity  assignment 
procedures  for  this  cost  function  as  well  as  the  logarithmic  cost  function 
and  the  linear  cost  function.  A  significant  result  issued  from  this  study 
tlirough  the  obscr/ation  that  the  average  message  delay  for  the  power  lav;  cost- 
function  could  very  closely  be  approximated  by  trie  average  message  delay  through 
the  system  constrained  by  a  linear  cost  function;  this  holds  true  in  the  case 
when  tne  . ^ stem  cost  is  held  fixed.  For  the  fixed  delay  case  we  found  that  the 
variation  of  the  system  cost  under  a  power  lav;  constraint  could  be  represented 
by  the  cost  variation  for  a  3 inear  cost  constraint  only  to  a  limited  extent. 

In  conjunction  with  pure  analytical  results  it  is  exeremeiy  useful  to  t;*ke 
advantage  of  system  simulation.  This  is  the  approach  we  describe  in  studying 
the  effect  of  rooting  procedures  and  comparing  methods  for  updating  these 
procedures.  We  indicated  that  a  synchro:  icus  moating  was  clearly  superior  to 


synchronous  updating  except  in  the  case  where  the  overhead  for  asynchronous 


updating  might  be  severe* . 


The  results  refer*red  to  above  serve  to  describe  the  behavior  of  computer 
network  systems  and  are  useful  in  the  early  stages  of  system  design ,  If  one 
is  desirous  of  obtaining  numerical  tools  for  choosing  the  precise  design 
parameters  of  a  system,  then  it  is  necessary  to  go  to  much  more  elaborate  ana¬ 
lytic  models  or  else  to  resort  to  efficient  search  procedures  (such  as  that 
described  in  Reference  *1)  in  order  to  locate  optimal  designs. 
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IlOST-lICm*  Communication 


INTRODUCTION 

The -Advanced  Research  Projects  Agency  (AHPA)  Computer*  Network  (hereafter 
referred  to  as  the  "AHPA  network"}  is  one  of  the  most  ambitious  computer 
networks  attempted  to  date . *  The  types  of  machines  and  operating  systems 
involved  in  the  network  vary  widely.  For  example,  the  computers  at  the  first 
four  sites  are  an  XDS  9^0  (Stanford  Research  Institute),  an  IBM  3o0/7b 
(University  of  California,  Santa  3ar’cara),  an  IDS  SIGMA-7  (University  of 
California,  Los  Angeles) ,  and  a  DEC  PDP-10  (University  of  Utah).  The  only 

k 

commonality  among  the  netv;ork  membership  is  the  use  of  highly  Interactive 
time-sharing  systems;  but,  of  course,  these  are  all  different  in  external 
appearance  and  implementation.  Furthermore,  no  one  node  is  in  control  cf 
the  rework.  This  has  insured  generality  and  reliability  but  complicates 
the  software. 

Of  the  networks  which  have  reached  the  op*  national  phase  and  been  re¬ 
ported  Ir  the  literature,  none  have  involved  the  variety  of  computers  and 
operating  systems  found  in  the  AHPA  network,  "or  example,  the  Carnegie- 
Kellori,  Princeton,  IBM  network  consists  of  360/67’s  with  identical  soft¬ 
ware.4"  Load  sharing  among  identical  batch  mac  lines  was  commonplace  at  Her* 
American  Rockwell  Corporation  in  the  early  1960’s.  Therefore ,  the  imple¬ 
mentors  of  the  piesent  network  have  been  only  slightly  influenced  by  earlier  ' 
network  attempts. 
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However,  early  time-sharing  studies  at  the  University  of  California  ai 
Berkeley,  MIT,  Lincoln  Laboratory,  and  System  Development  Corporation  (all 
ARPA  sponsored)  save  had  considerable  influence  or.  the  design  of  trie  networ. 
In  some  sense,  the  ARPA  network  of  time-shared  computers  is  a  natural  exaen- 
sion  of  earlier  time- sharing  concepts. 

H  e  network  is  seen  as  a  set  of  data  entry  and  exit  points  into  which 
individual  computers  insert  messages  destined  for  another  (or  same)  com¬ 
puter,  and  from  which  such  messages  emerge.  The  format  of  such  messages  and 
the  operation  of  the  network  was  specified  by  the  network  contractor  (EB&N) 
and  it  became  the  responsibility  of  representatives  of  the  various  computer 
sites  to  impose  such  additional  constraints  and  provide  such  protocol  as 
necessary  for  users  at  one  site  to  use  resources  at  foreign  sites.  This 
paper  details  the  decisions  that  have  been  made  and  the  considerations 
behind  these  decisions. 

Several  pec  pie  deserve  acknowledgment  ir.  this  effort.  J.  Rulifson  ar.-I 
W.  Duvall  of  BK3  participated  in  the  early  design  effort  of  the  protocol  a  id 
in  the  discussions  of  MIL.  C  ''•cloche  of  Thcnison-CS?  participated  in  the 
design  effort  wr ile  he  was  at  UCLA  and  provided  considerable  documentation. 
J.  .  of  Utah  and  P.  Kovner  of  Lincoln  Laboratory  reviewed  the  early  de¬ 
sign  and  NIL.  \K  Crowther  of  Bolt,  Beranek  end  Newman  contributed  the  ido  1 
of  a  virtual  net.  The  BB&N  staff  provided  substantial  assistance  and  guid¬ 
ance  while  delivering  the  network. 

We  have  found  that,  in  the  process  of  connecting  machines  and  operating 


systems  together,  a  great  deal  of  rapport  has  been  established  between  per¬ 
sonnel  at  the  various  network  node  sites.  Tie  resulting  mixture  of  ideas, 
discussions ,  disagreements ,  and  resolutions  has  been  highly  ro?roshin$  arm- 
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beneficial  to  all  involved,  and  we  regard  the  human  interaction  as  a  valuable 
by-product  of  the  main  effort. 


THE  NETWORK  AS  SEEN  BY  THE  HOSTS 


Before  going  on  to  discuss  operating  system  communication  protocol, 
some  definitions  <ire  needed. 

A  HOST  is  a  computer  system  v;h ich  is  part  of  the  network. 

An  IT  TP  (Interface  Message  Processor)  is  a  Honeywell  DDP-516  computer 
which  interfaces  with  up  to  four  HOSTs  at  a  particular  site,  and  allows  HOSTs 
access  into  the  network.  The  configuration  of  the  * nltial  four-HOST  network 
is  given  in  Figure  1.  The  IMPS  form  a  store -and- forward  communications  net¬ 
work.  A  companion  paper  in  these  proceedings  covers  the  IMPs  in  some  de¬ 
tail.^ 

A  message  is  a  bit  stream  less  than  8096  bits  long  which  is  given  to  ar 
IMP  by  a  HOST  for  trarisrnission  to  another  HOST.  The  first  32  bit  3  of  the 
message  arc  the  leader.  The  leader  contains  the  following  information: 

(a)  HOST 

(b)  Message  type 

(c)  Flags 

(d)  Link  number 


When  a  message  is  transmitted  from  a  HOST  to  its  IMP,  the  HOST  field  o/ 
the  leader  names  the  receiving  HOST.  When  the  message  .arrives  at  the  re¬ 
ceiving  HOST,  the  HOST  field  names  the  sending  HOST. 

Only  two  message  types  are  of  concern  'in  mils  paper.  Regular  moss  epos 
are  For.o  rated  by  a  HOST  and  sent  to  its  IMP  for  transmission  to  a  foreign 
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HOST.  The  other  rr.es sane  type  of  interest  is  a  R ;vK  (Request- for-! ’ext - 
Message ) .  RFNK*s  are  explained  in  conjunction  with  links. 

The  flap;  field  of  the  leader  controls  special  cases  not  of  concern  here. 
The  link  number  identifies  over  which  of  2 fb  logical  oaths  (links)  be¬ 


tween  the  sending  HOST  and  the  receiving  HOST  the  message  will  be  cent. 

Each  link  is  unidirectional  and  is  controlled  by  the  network  so  that  no  more 
than  one  message  at  a  time  may  be  sent  over  it.  This  control  is  implemented 
using.  RFNM  messages.  After  a  sending  HOST  has  sent  a  message  to  a  receiving 
HOST  over  a  particular  link,  the  sending  HOST  is  prohibited  from  sending 
another  message  over  that  same  link  until  the  sending  HOST  receives  a  RFHM. 
The  RFNM  is  generated  by  the  .IK?  connected  to  the  receiving  HOST,  and  the 
RFNM  is  sent  back  to  the  sending  HOST  after  the  message  has  entered  the  re¬ 
ceiving  HOST  It  is  important  to  remember  that  there  are  2p6  links  in  each 
direction  and  that  no  relationship  among  these  is  imposed  by  the  network. 

The  purpose  of  the  link  and  RFNM  mechanism  is  to  prohibit  individual 
users  from  overloading  an  IMP  or  a  HOST.  Implicit  in  this  purpose  is  the 
assumption  that  a  user  docs  not  use  multiple  links  to  achieve  a  wide  band, 
and  to  a  large  extent  the  HOST-HOST  protocol  cooperates  with  this  assumption 
An  even  more  basic  assumption,  of  course,  is  that  the  network* s  load  comes 
from  some  users  transmitting  sequences  of  messages  rather  than  many  users 
transmitting  single  messages  coinciiently. 

In  order  to  delimit  the  length  of  the  message,  and  to  make  it  easier 
for  HOSTs  of  differing  word  lengths  to  communicate,  the  following  formatting 
procedure  is  used.  When  a  HOST  prepares  a  message  for  output it  creates  a 
32-bit  leader.  Following  the  leader  is  a  binary  string,  called  markirr, 
consisting  of  an  arbitrary  number  of  zeroes,  followed  by.  a  one.  Marking 
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rrakes  it  possible  for  the  sending  HOST  to  synchronise  the  beginning  of  the 
text  of  a  message  with  its  word  boundaries .  When  the  last  bit  of  a  message 
has  entered  an  IMP,  the  hardware  interface  between  the  3ft?  and  HOST  appends 
a  one  followed  by  enough  zeroes  to  make  the  message  length  a  multiple  of  16 
bits.  These  appended  bits  are  called  padding.  Except  for  the  marking  and 
oaading,  no  limitations  are  placed  on  the  text  of  a  message.  Figure  2  shows 
a  typical  message  sent  by  a  24-bit  machine 


DESIGN  CONCEPTS 

The  computers  participating  in  the  network  are  alike  in  two  important 
respects:  each  supports  research  independent  cf  the  network,  and  each  is 
under  the  discipline  of  a  tJme-sharing  system.  These  facts  contributed  to 
the  following  design  philosophy. 

First,  because  the  computers  in  the  network  have  independent  purpose: 
it  is  necessary  to  preserve  decentralized  administrative  control  of  the 
various  computers.  Since  all  of  the  time-sharing  suoervisors  possess 
elaborate  and  definite  accounting  and  resource  lllocation  mechanisms,  we 
arranged  matters  so  that  these  mechanisms  would  control  the  load  due  to  t 
network  in  the  same  way  they  control  locally  generated  load. 

Second,  because  the  computers  are  all  operated  under  time-sharing 
disciplines,  it  seemed  desirable  to  facilitate  basic  interactive  mechanism::' . 

Third,  because  this  network  3s  used  by  experienced  programmers  it  was 
imperative  to  provide  the  widest  latitude  in  using  the  network.  Restriction 
concerning  character  sets,  programming  languages,  etc.  would  not  be  telerat 
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and  we  avoided  such  restrictions. 

Fourth,  again  because  the  network  is  used  by  experienced  programmers ; 
it  was  felt  necessary  to  leave  the  design  cn en-ended .  .  We  expect  that 
conventions  will  arise  from  time  to  time  as  experience  is  gained,  but  we 
felt  constrained  not  to  innose  then  arbitrarily. 

Fifth,  in  order  to  make  network  participation  comfortable,  or  in 
some  cases,  feasible,  the  software  interface  to  the  network  should  require 
minimal  surgery  on  the  HOST  operating  system. 

Finally,  we  accepted  the  assumption  stated  above  that  network  use 
consists  of  prolonged  converse * ons  instead  of  one-shot  requests. 

These  considerations  leu  the  notions  of  connections,  a  Network 
Control  Program,  a  control  link,  control  commands,  sockets,  and  virtual 
nets . 


A  connection  is  an  extension  cf  a  link.  A  connection  connects  two 
nrceesses  so  that  output  from  one  process  is  input  to  the  other.  Con¬ 
nections  are  simplex,  so  two  connections  are  needed  if  two  processes  are 
to  converse  in  both  directions. 

Processes  wjthin  a  HOST  communicate  with  tne  network  through  a 
*  Network  Control  Program  (NCP).  In  most  iiOSTs,  the  NCP  will  be  part  of 
the  executive,  sc  that  processes  will  use  system  calls  to  communicate 
with  it.  The  primary  function  of  the  NCP  is  to  establish  connections, 
break  connections ,  switch  connections,  and  control  flow. 

In  r-der  to  accomplish  its  tasks,  a  NCP  in  one  HOST  must  con  muni cate 


with  a  NCP  in  another  HOST, 
pair  of  HOSTs  has  been  cesi 


To  this  erd,  a  particular  link  between  each 
lated  as  the  control  link.  Messages  received 
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over  the  control  link  are  always  interpreted  by  the  NCP  as  a  sequence  of 
one  or  more  control  commands .  As  an  example,  one  of  the  kinds  of  control 
comriands  is  used  to  assign  a  link  and  initiate  a  connection,  while  another 
kind  carries  notification  that  a  connection  has  been  terminated.  A  par¬ 
tial  sketch  of  the  syntax  and  semantics  of  control  commands  is  given  in 
.  the  next  section. 

A  major  issue  is  how  to  refer  to  processes  in  a  foreign  HOST.  Each 
HOST  has  some  internal  naming  scheme,  but  these  various  schemes  often  are 
incompatible.  Since  it  is  not  practical  to  impose  a  corrsnon  internal 
process  naming  scheme,  an  intermediate  name  space  was  created  with  a 
separate  portion  of  the  name  space  given  to  each  HOST.  It  Is  left  to 
each  HOST  to  map  internal  process  identifiers  into  its  name  space. 

The  elements  of  the  name  space  are  ca3J.ec  sockets .  A  socket  forms 
•  one  end  of  a  connection,  and  a  connection  is  fully  specified  by  a  pair  of 
sockets.  A  socket  is  snecified  by  the  concatenation  of  three  numbers; 

(a)  a  user  number  (24  bits) 

(b)  a  HOST  number  (8  cits) 


•  (c)  All!  (8  bits) 

•  * 

'  A  typical  socket  is  illustrated  In  Figure  3* 

Each  HOST  in  assigned  all  sockets  in  the  name  space  which  have 
field  (b)  equal  to  the  HOST'S  own  identification. 

A  socket  is  either  a  receive  socket  or  a  send  socket,  and  Is  so 
marked  by  the  lev/- order  bit  of  the  AEiJ  (0  ?eeive,  1  -  send).  The 
other  seven  bits  of  the  AKN  simply  provide  a  sizable  population  of 
sockets  for  each  user  nurrher  at  each  HOST.  (API*  stands  for  "another 


eight-bit  number".) 
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Kach  user  is  assigned  a  24-bit  user  number  which  uniquely  identifies 
him  throughout  the  network.  Generally  this  will  be  the  8-bit  HOST  number 
of  his  borne  HOST,  followed  by  16  bits  which  uniquely  identify  him  at  that 
HOST,  Provision  can  also  be  made  for  a  user  to  have  a  user  number  not 
keyed  to  a  particular  HOST,  an  arrangement  desirable  for  mobile  users  who 
might  have  no  hone  HOST  or  more  than  one  home  HOST.  Tnis  24-bit  user 
number  is  then  used  in  the  following  manner.  When  a  user  signs  onto  a 
HOST,  his  user  number  is  looked  up.  Thereafter,  each  process  the  user 
creates  is  tagged  with  his  user  number.  When  the  user  signs  onto  a 
foreign  HOST  via  the  network,  his  same  user  number  is  used  to  tag  proc¬ 
esses  he  creates  in  that  HOST.4  The  foreign  HOST  obtains  the  user  number 
either  by  consulting  a  table  at  login  time,  as  the  heme  HOST  does,  or  by 
noticing  the  identification  of  the  caller.  The  effect  of  propagating  the 
user's  number  is  that  each  user  creates  his  own  virtual  net  consisting  of 
processes  he  has  created.  This  virtual  net  may  span  an  arbitrary  number 
of  HOSTs.  It  will  thus  be  easy  for  a  user  to  connect  his  processes  in 
arbitrary  ways,  while  still  permitting  him  to  connect  his  processes  with 
those  in  other  virtual  nets. 

The  relationship  between  sockets  and  processes  is  now  describable 
(see  Figure  4).  For  each  user  number  at  each  HOST,  there  are  128  send 
sockets  and  128  receive  sockets,  A  process  may  request  from  the  local 
HOP  the  use  of  any  one  of  the  sockets  with  the  same  user  number;  the  re¬ 
quest  is  granted  if  the  socket  is  not  otherwise  in  use.  The  key  observa¬ 
tion  here  is  that  a  socket  requested  ay  a  process  cannot  already  be  in 


9 


1 10ST-I  IQST  Communication 


r- 

i 

1 

§ 

S 

s 

i 


such  a  process  is  controlled  by  the  same  user. 

An  unusual  aspect  of  the  HOST-HOST  protocol  is  that  a  process  may 
switch  its  end  of  a  connection  from  one  socket  to, another.  Hie  new  socket 
may  be  in  any  virtual  net  and  at  any  HOST,  ard  the  process  may  initiate  a 
switch  either  at  the  time  the  connection  is  fceing  established,  or  later. 
The  most  general  forms  of  switching  entail  quite  complex  implementation, 
and  are  not  germane  to  the  rest  of  this  paper,  so  only  a  limited  form  . 
will  be  explained.  This  limited  form  of  switching  provides  only  that  a 
process  may  substitute  one  socket  for  another*  while  establishing  a  con¬ 
nection.  The  new  socket  must  have  the  same  user  number  and  HOST  number, 

4 

and  the  connection  is  still  established  to  the  same  process.  This  form 
of  switching  is  thus  only  a  way  of  relabelling  a  socket,  for  no  change 
in  the  routing  of  messages  takes  place.  In  the  next  section  we  document 
the  system  calls  and  control  commands;  in  the  section  after  next,  we  con¬ 
sider  how  login  might  be  implemented. 


SYSTEM  CALLS  AMD  CONTROL  CGf-tfANDS 

Here  we  sketch  the  mechanics  of  establishing,  switching  and  breaking 
a  connection.  As  rioted  above,  the  NCP'  interacts  with  user  processes  via 
system  calls  ard  with  other  NC?s  via  control  commands.  We  therefore  be¬ 
gin  with  a  partial  description  of  system  calls  and  control  commands. 

System  calls  will  vary  from  one  operating  system  to  another,  so  the 
following  description  is  only  suggestive.  We  assume  here  that  a  process 


has  several  Input-output  paths  which  v:c 


will  call 


Each  port  ray  be 
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connected  to  a  sequential  I/O  device,  and  while  connected,  transmits 
information  in  only  one  direction.  We  further  assume  that  the  process  is 
blocked  (dismissed,  slept)  v;hile  transmission  proceeds.  The  following  is 
the  list  of  system  calls: 

Init  <port>,  <AKN  1>,  <AEN  2>,  < foreign  socket > 
where  <port>  is  part  of  the  process  issuing  the  Init 
<AEN  1>  ) 

and  >  are  8-bit  AEN*s  (see  Figure  3) 

<AEN  2>  ) 

The  first  AEN  is  used  to  initiate  the  correction;  the  second  is 
used  while  the  connection  exists. 

<foreign  socket >  is  the  JJO-bit  sockc;  name  of  the  distant  end 
of  the  connection. 

X 

X 

■> 

The  low-order  bits  of  <AEN  1  >and  <A3lN  2>  must  agree,  and  these 
must  be  the  complement  of  the  low-order  bit  of  <foreign  socket> 

The  NCP  concatenates  <AKN  1>  and  2>  each  with  the  user 
number  of  the  process  arid  the  HOST  n^nber  to  fonn  0-bit  sockets 
It  then  sends  a  Request  for  Connection  (RFC)  control  command  to 
the  distant  NCP.  When  the  distant  NCP  responds  positively,  the 
connection  is  established  and  the  process  is  unblocked.  If  the 
distant  NCP  responds  negatively,  the  local  NCP  unblocks  the 
requesting  process,  but  informs  it  that  the  system  call  has 
failed. 


11 


HOST-LOST  Communication 


Liston  <port>,  <AEN  1> 

whore  <port>  and  <AEN  1>  are  as  above.  The  NCP  retains  the  ports 

and  <AI:M  1>  and  blocks  the  process.  When  ar.  RFC  control  command 
•arrives  naming  the  local  socket,  the  process  is  unblocked  and 
notified  that  a  foreign  process  is  calling. 

Accept  <AEN  2> 

After  a  Listen  ha.*,  been  satisfied,  the  process  rray  either 
refuse  the  call  or  accept  it  and  switch  it  to  another  socket. 

To  accept  the  call,  the  process  issues  the  Accept  system  call. 
The  NCP  then  sends  b^ck  an  RFC  control  command. 

Close  <port> 

After  establishing  a  connection,  a  process  issues  a  Close 
to  break  the  connection.  The  Close  is  also  issued  after  a 
Listen  to  refuse  a  call. 

Transmit  <nort>,  <addr> 

If  <oort>  is  attached  to  a  send  socket,  <addr>  points  to 
a  message  to  be  sent.  This  message  is  preceded  by  its  length 
in  bits. 

If  <port>  is  attached  to  a  receive  socket,  a  message  is 
stored  at  <addr>.  Tne  length  of  the  message  is  stored  first. 
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Control  conirands 

A  vocabulary  of  control  commands  has  been  defined  fcr  communication 
between  Network  Control  Programs,  fad;  control  command  consists  of  an 
vS-blt  operation  cede  to  indicate  its  function,  followed  by  some  parame¬ 
ters.  The  number  and  format  of  parameters  is  fixed  for  each  operation 
code.  A  sequence  of  control  commands  destined  for  a  particular  HOST  can 
be  packed  into  a  single  control  message. 

RFC  <rr\y  socket  1>,  <my  socket  2>, 

<your  socket >,  (<link>) 

This  command  is  sent  because  a  process  has  executed  either  an  Init 

4 

system  call  or  an  Accept  system  call.  A  link  s  assigned  by  the  prospec¬ 
tive  receiver,  so  it  is  omitted  if  socket  3>  is  a  send  socket. 

There  is  distinct  advantage  in  using  the  same  commands  both  to 
initiate  a  connection  (Init)  and  to  accept  a  call  (Accept).  If  the  re¬ 
sponding  command  were  different  from  the  initiating  command,  then  tv/o 
processes  could  call  each  other  and  become  blocked  waiting  for  each  other 
to  respond.  With  this  scheme  no  deadlock  occui'S  and  It  provides  a  more 
compact  way  to  connect  a  set  of  processes. 

CLS  <my  socket >,  <your  socket > 

The  specified  connection  is  terminated 
CEASE  <link> 

When  the  receiving  Drocess  does  not  consume  its  input  as  fast  as  It 
arrives,  the  buffer  space  in  the  receiving  HOST  is  used  to  queue  the 
waiting  messages.  Since  only  limited  space  is  generally  available,  the 
receiving  HOST  may  need  to  inhibit  the  sending  HOST  from  sending  any  more 
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messages  over  the  offending  connection.  Vihen  the  sending  HOST  receives 
this  conmand,  it  ray  block  the  process  generating  the  messages. 

RESUME  <link> 

This  command  is  also  sent  from  the  receiving  HOST  to  the  sending 
HOST  and  negates  a  previous  CEASE. 

LOGGING  IN 

W?  assume  that  withir  each  HOST  there  is  always  a  process  in  execu¬ 
tion  which  listens  to  login  requests.  We  call  this  process  the  logger, 
and  it  is  part  of  a  special  virtual  net  whose  user  number  is  zero.  The 
logger  is  programed  to  listen  to  calls  on  socket  number  0.  UDon  receiv¬ 
ing  a  call,  the  logger  switches  it  to  a  higher  (even)  numbered  sockets, 
and  returns  a  csll  to  the  socket  numbered  one  less  than  the  send  socket 
originally  calling.  In  this  fashion,  the  logger  can  initiate  127  conver¬ 
sations  . 

To  illustrate,  assume  a  user  whose  identification  is  X* 010005’  (user 
number  5  at  UCLA)  signs  into  UCLA,  starts  up  one  of  his  programs,  and 
this  program  wants  to  start  a  process  at  SHI.  Mo  process  at  SRI  except 
the  logger  is  currently  willing  to  listen  to  cur  user,  so  he  executes 
Init,  <port>  =  1,  <AEN  1>  =  '[ t  <AEN  2>  =  7, 

< foreign  socket >  =  0. 

His  process  is  blocked,  and  the  NCP  at  UCLA  sends 
Ri’C  <my  socket  1>  =  X  ’  0100050107  ’ . 

<my  socket  2>  =  X‘ 0100050107’ , 

<your  socket >  =  X *0000000200* 
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The  lcjwpr  at  SHI  is  notified  when  this  rr.esssEe  is  received,  because  it 

has  previously  executed 

Lister,  <pcrt>  =  9,  <AKN  i>  =  0. 

Tne  logger  then  executes 

Accept  <AM'I  2>  “  88. 

In  response  to  the  Accept ,  the  SHI  NCP  sends 

pj?Q  <my  socket  1>  =  X!OOOOOC02QO 

<my  socket  2>  =  X*00000C0258* 

<your  socket >  =  X1 0100050107* 

<link>  =  37 

where  the  link  has  been  chosen* from  the  set  cf  available  links.  The  SRI 

logger  then  executes 

Init  <port>  =  10 

<AEN  1>  =  89,  <AEN  2>  =  89, 

<foreifyi  socket >  =  X*01C0050106 

which  causes  the  NCP  to  send 

PPC  erry  socket  1>  -=  X* 00000(0259 

<rny  socket  2>  =  x* 00000(0259 

<your  socket >  =  X *0100050106* 

me  process  at  UCLA  is  unblocked  am  notifiec  of  the  successful  Init. 
Because  the  SRI  looser  always  initiates  a  correction  to  the  AEN  one  less 
.than' it  has  just  been  connected  to,  the  UCLA  process  then  executes 

Listen  <port>  =  11 
<AEN  1>  =  C 
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and  v;hen  unblocked, 

Accept  <AEN  2>  =  6. 

When  these  transactions  are  complete,  she  UCL-\  process  Is  doubly  corrected 
to  the  logger  at  Shi .  The  logger  will  then  interrogate  the  UCLA,  process, 
and  if  satisfied,  create  a  new  process  at  SRI.  This  new  process  will  be 
tagged  with  the  user  number  X’ 010005’,  and  both  connections  will  be 
si  ~ched  to  the  new  process.  In  this  case,  switching  the  coanections  to 
the  new  process  corresponds  to  "passing  the  console  down"  in  many  time¬ 
sharing  systems. 

USER  LEVEL  SOFTWARE 

At  the  user  level,  subroutines  which,  marage  data  buffers  and  format 
input  destined  for  other  SCSI's  are  provided.  It  is  not  mandatory  that 
the  user  use  such  subroutines ,  since  the  user  has  access  to  the  network 
system  calls  in  his  monitor. 

In  addition  to  user  programming  access,  it  is  desirable  to  have  a 
subsystem  program  at  each  HOLT  which  makes  the  network  immediately  acces¬ 
sible  from  a  teletype-like  device  without  special  programming.  Subsystems 
are  commonly  used  system  components  such  as  text  editors,  compilers  and 
interpreters.  An  example  of  a  network-related  subsystem  is  TELNET,  which 
will  allow  us eio  at  the  University  of  Utah  to  connect  to  Stanford  Research 
Institute  and  arse or  as  regular  terminal  users.  It  is  expected  that  more 
sophisticated  subsystems  will  be  developed  in  time,  but  this  basic  one 
will  render  the  early  network  immediately  useful. 

A  user  at  the*  University  of  Utah  (UTAH)  Is  sitting  at  a  teletype 
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dialed  into  the  University’s  PDP-10/90  time- sharing  system.  He  wishes  to 
operate  the  Conversational  Algebraic  Language  (CAL)  subsystem  on  the 
XUS- 9^0  at  Stanford  Res*: ' *v;h  Institute  (SRI)  in  i’enlo  Park,  California. 

A  typical  TETNH?  dialog  illustrated  in  Figure  5*  The  meaning  of  each 
line  of  dialog  :.s  discussed  here. 

(i)  The  user  signs  in  at  UTAH 

(ii)  The  PDP-10  run  command  stains  up  the  TELNET  subsystem 
at  the  user’s  HOST. 

(iii)  The  user  identifies  a  break  character  which  causes  ary 
message  following  the  break  to  be  interpreted  locally 
rather  than  being  sent  on  to  the  foreign  HOST. 

(iv)  The  TELNET  subsystem  will  rake  the  appropriate  system 
calls  to  establish  a  pair  of  connections  to  the  SHI 
logger.  The  connections  will  be  established  only  if 
SRI  accepts  another  foreign  user. 

The  UTAH  user  is  now  In  the  pre-logged-in  state  at  SRI.  This  is  analogou 
to  the  standard  teletype  user’s  state  after  dialing  into  a  computer  and 
making  a  connection  but  before  typing,  anything. 

(v)  The  user  signs  in  to  SRI  with  a  standard  login  command. 
Characters  typed  on  the  user’s  teletype  are  transmitted  unaltered  through 
the  PDP-10  (user  HOST)  and  on  to  the  9^0  (serving  HOST) .  The  PDP-10 
TELNET  subsystem  will  have  automatically  switched  to  full-duplex, 
character-by-character  transmission,  since  this  is  required  by  SRI’s  9^0* 
Full  duplex  operation  is  allowed  for  by  the  PDP-10,  though  not  used  by 
most  Digital  Equipment  Corporation  subsystems. 
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(vi)  and  (vii)  The  9iJd  subsystem,  CAL,  Is  started. 

At  this  point,  the  user  wishes  to  lead  a  CAL  file  into  the  940  CAL  sub¬ 
system  from  the  file  system  on  his  local  PDP-10 . 

(viii)  CAL  is  instructed  to  establish  a  connection  to  UTAH  in 
•order  to  receive  the  file.  "NEIVJRK"  is  a  predefined 
9^0  name  similar  in  nature  to  "PAPER  TAPE"  or  "TELETYPE", 
(ix)  Finally,  the  user  types  the  break  character  (#)  followed 
by  a  command  to  his  PDP-10  TELNET  program,  which  sends 
the  desired  file  to  SRI  from  Utah  on  the  connection 
just  established  for  this  Durpose.  Hie  user’s  next 
statement  is* in  CAL  again. 

The  TELNET  subsystem  coding  should  be  minimal  for  it  i:  essentially 
a  shell  program  built  over  the  network  system  calls.  It  effectively 
established  a  shunt  in  the  user  HOST  between  the  remote  user  and  a  dis¬ 
tant  serving  HOST. 

Given  the  basic  system  primitives,  the  TELNET  subsystem  at  the  user 
HOST  and  a  manual  for  the  serving  HOST,  the  network  can  be  profitably 
employed  by  remote  useis  today. 

HIGHER  LEVEL  PROTOCOL 

The  network  poses  special  problems  where  a  high  degree  of  inter¬ 
action  is  required  between  the  user  and  a  particular  subsystem  on  a 
foreign  HOST.  These  problems  arise  due  to  heterogeneous  consoles,  local 
operating  system,  overhead,  and  network  transmission  delays.  Unless  we 
use  special  strategies  it  may  be  difficult  cr  even  impossible  for  a  distant 
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user  to  make  use  cf  the  more  soohisticated  subsystems  offered.  While 
these  difficulties  are  especially  severe  in  the  area  of  graphics,  pro^Ie:  . 
may  arise  even  for  teletype  interaction.  For  example,  suppose  that  a 
foreign  subsystem  is  designed  for  teletype  consoles  connected  by  tele¬ 
phone,  and  then  this  subsystem  becomes  available  to  network  users.  This 
subsystem  might  have  the  following  characteristics. 

1.  Except  for  echoing  and  correction  of  mistyping,  no  action 
is  taken  until  a  carriage  return  in  typed. 

2.  All  characters  except  V1,  and  carriage  return  are 
echoed  as  the  character  typed. 

3.  «-  causes  deletion  of  the  immediately  preceding  accepted 
character,  and  is  echosed  as  that  character. 

4.  t  causes  all  previously  typed  characters  to  be  ignored.  A 
carriage  return  and  line  feed  are  echoed. 

5.  A  carriage  return  is  echoed  as  a  carriage  return  followed 
by  a  line  feed. 

If  each  character  typed  is  sent  in  its  own  message,  then  the  characters 

HELLO^P  c.r. 

cause  nine  messages  in  each  direction.  Furthermore,  each  character  is 
*  handled  by  a  user  level  program  in  the  local  HOST  before  being  sent  to 
the  foreign  HOST. 

Now  it  is  clear  that  if  this  particular  example  were  Important,  we 
would  quickly  implement  rules  1  to  5  in  a  local  HOST  program  and  send 
only  complete  lines  to  the  foreign  HOST.  If  the  foreign  HOST  program 
could  not  be  modified  so  as  to  not  generate’ echoes,  then  the  local  prepram 
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could  not  only  echo  properly,  it  could  also  throw  away  the  later  echoes 
from  the  foreirn  HOST.  However,  t:.e  problem  is  net  any  particular  inter¬ 
action  scheme;  the  problem  is  that  we  e>qpect  nany  of  these  kinds  of 
schemes  to  occur.  We  ..ave  not  found  any  general  solutions  to  these  prob¬ 
lems,  but  some  observations  and  conjectures  may^ lead  the  way. 

With  respect  to  heterogeneous  consoles,  we  note  that  although  con¬ 
soles  are  rarely  compatible,  many  are  equivalent.  It  is  probably  reason¬ 
able  to  treat  a  model  37  teletype  as  the  equivalent  of  an  IBM  27^1. 
Similarly,  most  storage  scopes  will  form  an  equivalence  class,  and  most 
refresh  display  scopes  will  form  another.  Furthermore,  a  hierarchy  might 
emerge  with  members  of  one  class  usable  in  place  of  those  in  another,  but 
not  vice  versa.  We  can  imagine  that  any  scope  might  be  an  adequate  sub¬ 
stitute  for  a  teletype,  but  hardly  the  reverse.  This  observation  leads 
us  to  wonder  if  a  network-wide  language  for  consoles  might  be  possible . 
Such  a  language  would  provide  for  distinct  treatment  of  different  classes 
of  consoles,  with  semantics  appropriate  to  each  class.  Each  site  could 
then  write  inter  face  programs  for  its  consoles  to  make  them  look  like 
network  standard  devices. 

Another  observation  is  that  a  user  evaluates  an  interactive  system 
by  comparing  the;  speed  of  the  system fs  responses  with  his  own  expecta¬ 
tions.  Sometimes  a  user  feels  that  he  has  made  only  a  minor  request,  so 
the  response  should  be  Immediate;  at  other  times  he  feels  he  has  made  a 
substantial  request,  and  is  therefore  willing  to  wait  for  the  response. 
Some  interactive  subsystems  are  especially  pleasant  to  use  because  a 
great  deal  of  work  has  gone  into  tailoring  the  responses  to  the  userfs 
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expectations.  In  the  network,  however,  a  local  user  level  process  inter¬ 
venes  between  a  local  console  and  a  foreign  subsystem,  and  we  may  expect 
the  response  time  for  minor*  requests  to  degrade.  Nov;  it  nay  happen  that 
all  of  this  tailoring  of  the  interaction  is  fairly  independent  of  the 
portion  of  the  subsystem  which  does  the  heavy  computing  or  I/O.  In  such 
a  case,  it  may  be  possible  to  separate  a  subsystem  into  two  sections. 

One  section  would  be  the  "substantive”  portion;  the  other  would  be  a 
"front  end”  which  formats  output  to  the  user,  accepts  his  inputs,  and 
controls  computationally  simple  responses  such  as  echoes.  In  the  example 
above,  the  program  to  accumulate  a  line  and  generate  echoes  would  be  the 
front  end  of  same  subsystem.  V/e  now  take  notice  of  the  fact  that  the 
local  HOSTs  ha^e  substantial  computational  power,  tut  our  current  designs 
make  use  of  the  local  HOST  only  as  a  data  concentrator.  This  is  somewnat 
ironic,  for  the  local  HOST  is  not  only  poorly  utilized  as  a  data  concen¬ 
trator,  it  also  degrades  performance  because  of  the  delays  it  introduces. 

These  arguments  have  led  us  to  consider  the  possibility  of  a  Network 
Interface  Language  (NIL)  which  would  be  a  network-wide  language  for 
writing  the  front  end  of  Interactive  subsystems.  This  language  would 
have  the  feature  that  subprograms  communicate  through  network-like  con¬ 
nections.  The  strategy  is  then  to  transport  the  source  code  for  the 
front  end  of  a  subsystem  to  the  local  HOST,  where  it  would  be  compiled 
and  executed . 

During  preliminary  discussions  we  have  agreed  that  MIL  should  have 
at  least  the  following  semantic  properties  not  generally  found  in 
languages . 
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l’  Concurrency.  Because  messages  arrive  asynchronously  on 
different  connections,  and  because  user  input  is  not 
synchronized  with  subsystem  output,  NIL  must  include 
semantics  to  accurately  model  the  possible  concurrencies . 

2.  Program  Concatenation.  It  Is  very  useful  to  be  able  to 
insert  a  program  in  between  two  other  programs.  To  achieve 
this,  the  interconnection  of  programs  would  be  specified 

at  run  time  and  would  not  be  iinplicit  in  the  source  code. 

3. ’  Device  substitutability.  *  It  is  usual  to  define  languages 

so  that  one  device  may  be  substituted  for  another.  The 
requirement  here  is  that  any  device  can  be  modelled  by  a 
NIL  program.  For  example,  if  a  network  standard  display 
controller  manipulates  tree-structures  according  to  mes¬ 
sages  sent  to  it  then  these  structures  mast  be  easily 
imnlementable  in  NIL. 

NIL  has  not  been  fully  specified,  and  reservations  have  been  expressed 

about  its  usefulness.  Those  reservations  hirge  upon  our  conjecture  that 

it  Is  possible  to  divide  an  interactive  subsystem  into  a  transrx>rtable 

front  end  which  satisfies  a  user’s  expectations  at  low  cost  and  a  more 

substantial  stay-at-home  section.  If  our  cor jecture  is  false,  then  NIL 

will  not  be  useful;  otherwise  It  seems  worth  pursuing.  Testing  of  this 

* 

conjecture  and  further  development  of  NIL  will  take  priority  after  low 
level  IIOST-HO S*  protocol  has  stabilized. 
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HOST/IMP  IKTCRFACi^r. 

The  hardware  and  software  interfaces  between  HOST  and  IMP  is  an 
area  of  particular  concern  '  ;ho  HOST  organizations .  Considering  the 
diversity  of  HOST  computers  to  which  a  standard  IMP  must  connect,  the 
hardware  interface  was  made  bit  serial  and  full-duple)' .  2a ch  HOST  or- 
ganizatic  1  implements  its  half  of  tnis  very  simple  interface. 

The  software  interface  is  equally  simple  and  consists  wf  messages 
passed  back  and  forth  between  the  IMP  and  HOST1  programs.  Special  error 
and  signal  messages  are  defined  as  well  as  messages  containing  normal 
data.  Messages  waitirg  in  queues  in  either  machine  are  sent  at  the 
pleasure  of  the  machine  in  v/Men  they  reside  with  no  concern  for  the  needs 
of  the  other  computer. 

The  effect  of  the  present  softv/are  interface  is  the  needless  re¬ 
buffer  of  all  messages  in  the  HOST  in  addition  to  the  buffering  in 
the  IMP.  The  messages  have  no  particular  order  other  than  arrival  times 
at  the  IMP.  The  Network  Control  Program  at  one  HOST  (e.g.,  Utah)  jjeeds 
waiting,  RFNM’s  before  all  other  messages.  At  anoth.er  site  (e.g. ,  SRI), 
the  NCP  could  benefit  by  receiving  messages  fer  the  user'  who  is  next  to 
be  run. 

What  is  needed  is  coding  representing  the  specific  needs  of  the  HOST 
on  both  sides  of  the  Interface  to  make  intelligent  decisions  about  what 
to  transmit  next  over  the  channel.  With  the  present  software  interface 
the  channel  in  one  direction  once  cornel  tted  to  a  particular  message  is 
then  locked  up  for  un.to  80  milliseconds !  This  approaches  one  teletype 
character  time  and  needlessly  limits  full-duplex,  character  by  character, 
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interaction  over  the  net.  At  the  very  least,  the  IKP/HGST  protocol 
shoiud  be  expanded  to  permit  each  side  to  assist  the  other  in  scheduling 
messages  over  the  channels. 

CONCLUSIONS 

At  this  time  (February  1970)  the  initial  network  of  four  sites  is 
just  beginning  to  beginning  to  be  utilized.  The  conromicat ions  system 
of  four  IMPs  and  wide  band  telephone  lines  have  been  operational  for  two 
months.  Programmers  at  UCLA  have  signed  in  as  users  of  the  SRI  9^0.  More 

significantly,  one  of  the  authors  (S.  Carr)  living  in  Palo  Alto  uses  the 

* 

Sal  Lake  PDP-10  on  a  daily  basis  by  first  connecting  to  SRI.  We  thus 
have  first  hand  experience  that  remote  interaction  is  possible  and  is 
highly  effective. 

V/ork  on  the  ARPA  network  has  generated  new  areas  of  interest.  NIL 
is  one  example,  and  interprocess  communication  is  another.  Interprocess 
communication  over  the  network  is  a  subcase  cf  general  interprocess  com¬ 
munication  in  a  nultlprop3rarcr.ed  environment .  The  mechanism  of  connec¬ 
tions  seems  to  be  new,  and  we  wonder  whether  this  mechanism  is  useful 
even  when  the  piocesses  are  within  the  same  computer. 
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fipiire  2  A  typical  message  from  a  2^— bit  machine 


connection 


relationship  between  sockets  and  processes 


(i)  .LOGIN© 

(ii)  .R  TELNET© 

(iii)  ESCAPE  CHARACTER  IS  #© 

(iv)  CONNECT  TO  SRI© 

(V)  ©ENTER  CARR.© 

(Vi)  ©CAL.© 

(Vii)  CAL  AT  YOUR  SERVICE© 

(viii)  >READ  FILE  FROM  NETWRK.© 
(ix)  *NETWRI<g<3-DSKsMYFILE.CA'-© 


Figure  5  A  typical  TELNET  dialog 
Underlined  characters  are  those  typed  by  the  user 


